Input Configuration

---

Filebeat supports two approaches to log collection: filebeat.inputs for manually specifying file paths, and filebeat.modules for pre-configured collection from common applications. The two can be used together in the same config.

---

filebeat.inputs — File Path Inputs

Use filebeat.inputs when collecting from custom or non-standard log paths, or when no module exists for the application.

log / filestream Type

The primary input type for reading lines from log files. filestream is the modern replacement for log (introduced in v7.16) — prefer filestream on Filebeat 7.16+.

filebeat.inputs:

  # Single file
  - type: log
    paths:
      - /var/log/auth.log

  # Wildcard — all .log files in a directory
  - type: log
    paths:
      - /var/log/application/*.log

  # Multiple paths in one input
  - type: log
    paths:
      - /var/log/system.log
      - /var/log/wifi.log

Useful log/filestream Options

Option	Description	Example
paths	List of glob patterns to collect	- /var/log/*.log
encoding	File encoding	utf-8, utf-16be-bom
exclude_lines	Regex — drop lines matching the pattern	['^DEBUG']
include_lines	Regex — only collect lines matching the pattern	['^ERR', '^WARN']
exclude_files	Regex — skip files whose path matches	['.gz$']
ignore_older	Skip files not modified within this duration	72h
scan_frequency	How often Filebeat scans for new/updated files	10s
tags	Tags to append to all events from this input	[webserver, prod]
fields	Custom key-value fields to add to events	env: production

- type: log
  paths:
    - /var/log/nginx/access.log
  ignore_older: 168h
  exclude_lines: ['^127\.0\.0\.1']
  tags: [nginx, webserver]
  fields:
    server_role: reverse_proxy

Filebeat Does Not Support Network Shares

Filebeat cannot read from UNC paths, SMB shares, or cloud storage buckets using the log type. For cloud sources use the appropriate cloud input type (aws-s3, azure-eventhub, etc.).

---

tcp / udp Type

Listens on a local port for log streams sent over the network. Useful for collecting syslog from network devices (firewalls, switches, routers) or other systems that support syslog forwarding.

filebeat.inputs:
  - type: udp
    host: "0.0.0.0:514"
    max_message_size: 64KiB

  - type: tcp
    host: "0.0.0.0:1514"
    max_message_size: 64KiB

Option	Description
host	IP:Port to listen on. Use 0.0.0.0 to listen on all interfaces.
max_message_size	Maximum message size. Default is 10KiB. Increase for verbose logs.
read_buffer	UDP read buffer size.
timeout	Read/write timeout for socket operations.

Collecting Syslog from Network Devices

For firewalls, switches, or other network devices that support syslog forwarding, configure the device to forward syslog to the Filebeat host's IP on port 514 (UDP) or a custom port. Pair with the add_fields processor to tag events by source device.

---

filebeat.modules — Pre-Built Module Inputs

Filebeat modules provide pre-configured file paths, parsing, and ECS field normalization for common applications. Enabling a module is simpler than building a raw filebeat.inputs config and produces better-structured events for Kibana dashboards.

Enabling Modules

Modules can be enabled in filebeat.yml directly:

filebeat.modules:
  - module: zeek
    connection:
      enabled: true
    dns:
      enabled: true
    http:
      enabled: true

Or via the command line (creates a .yml file in modules.d/):

filebeat modules enable zeek suricata system auditd

Overriding Module File Paths

Each module has default file paths for the target OS. If logs are in a non-standard location, override the path using var.paths:

filebeat.modules:
  - module: apache
    access:
      enabled: true
      var.paths: ["/custom/path/httpd/access_log*"]
    error:
      enabled: true
      var.paths: ["/custom/path/httpd/error_log*"]

---

Multiline Log Handling

Some applications write a single logical event across multiple lines (Java stack traces, Python tracebacks, multiline JSON). Use the multiline option to merge these into a single Filebeat event.

- type: log
  paths:
    - /var/log/application/app.log
  multiline:
    type: pattern
    pattern: '^\d{4}-\d{2}-\d{2}'   # New event starts with a date (YYYY-MM-DD)
    negate: true                      # Lines NOT matching the pattern...
    match: after                      # ...are appended to the previous line

Option	Values	Description
type	pattern, count, while_pattern	Matching strategy
pattern	Regex	Pattern to match against
negate	true/false	Invert the pattern match
match	before, after	Append non-matching lines before or after the matching line
max_lines	Integer	Maximum lines per merged event (default: 500)
timeout	Duration	Flush incomplete multiline events after timeout (default: 5s)

Common Multiline Patterns

# Java stack traces (lines starting with whitespace or 'at' belong to previous event)
pattern: '^[[:space:]]|(^at )'
negate: false
match: after

# Python tracebacks (continuation lines are indented)
pattern: '^\s'
negate: false
match: after

# Multiline JSON (closing brace starts a new event)
pattern: '^\}'
negate: false
match: before

---

Global Processors

Apply transformations to all events regardless of input:

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - drop_fields:
      fields: ["agent.ephemeral_id", "ecs.version"]
      ignore_missing: true

---